feat: Add ContainerBackend with Docker and Podman #119

Fiona-Waters · 2025-10-07T15:03:29Z

What this PR does / why we need it:

This PR introduces a unified ContainerBackend that automatically detects and uses either Docker or Podman for local training execution. This replaces the previous separate LocalDockerBackend and LocalPodmanBackend implementations with a single, cleaner abstraction. You can see the Docker and Podman implementations in separate commits.

This implementation tries Docker first, then falls back to Podman if Docker is unavailable. This can be overridden via ContainerBackendConfig.runtime to force a specific runtime ("docker" or "podman"). An error is raised if neither runtime is available.
Unit tests for the backend implementation have also been added. Examples for using Docker and Podman will be added to the Trainer repo later.

Manually testing on Mac I had to specify the container_host like so:
Docker via Colima container_host=f"unix://{os.path.expanduser('~')}/.colima/default/docker.sock"
Podman Desktop container_host=f"unix://{os.path.expanduser('~')}/.local/share/containers/podman/machine/podman.sock"

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):

Fixes ##114 and #108

Checklist:
I need to look at adding docs. A README has been included.

Docs included if any changes are user facing

andreyvelich

Thank you for this @Fiona-Waters!
As we discussed here: #111 (comment), can we consolidate Podman and Docker under single container backend ?
Given that those backend should have similar APIs, I think it would be better to consolidate them, similar to KFP: https://www.kubeflow.org/docs/components/pipelines/user-guides/core-functions/execute-kfp-pipelines-locally/#runner-dockerrunner

Fiona-Waters · 2025-10-08T08:13:30Z

Thank you for this @Fiona-Waters! As we discussed here: #111 (comment), can we consolidate Podman and Docker under single container backend ? Given that those backend should have similar APIs, I think it would be better to consolidate them, similar to KFP: https://www.kubeflow.org/docs/components/pipelines/user-guides/core-functions/execute-kfp-pipelines-locally/#runner-dockerrunner

Thanks @andreyvelich I will look at updating the implementation.

Fiona-Waters · 2025-10-08T10:19:46Z

@andreyvelich @astefanutti regarding comments on this PR and #111 this is what I propose:

We have 3 backends:

Kubernetes
Subprocess
Local Container

For the Local Container backend we automatically try Docker first, then Podman and then fallback to Subprocess if neither runtime is available. We use the adapter pattern with a container client adapter unified interface, and docker and podman specific calls are implemented in separate adapter classes.
There could also be an option where users can force a specific runtime for example:
LocalContainerBackendConfig(runtime="docker")
This implementation will make it easy to add support for other container runtimes in the future, if thats a possibility.
Please let me know what you think. Thanks!
cc @briangallagher

andreyvelich · 2025-10-08T13:41:03Z

Sure, that looks great @Fiona-Waters!

fallback to Subprocess if neither runtime is available

Why do we need to fallback to subprocess ?
I would imagine we have 3 backend support, and user decide what they want to use:

KubernetesBackend()
ContainerBackend()
LocalProcessBackend()

In the ContainerBackend users can select:

 ContainerBackend(
  ContainerBackendConfig(container_runtime="docker")
)
or
 ContainerBackend(
  ContainerBackendConfig(container_runtime="podman")
)

astefanutti · 2025-10-08T14:07:34Z

@Fiona-Waters that sounds to me. I agree the fallback logic may really apply to choose the default container runtime.

Other than that, I'd be inclined to drop the "Local" prefix entirely. Even Kubernetes could run local with KinD, and I doubt the SDK will ever do remote process.

Fiona-Waters · 2025-10-08T14:18:37Z

Sure, that looks great @Fiona-Waters!

fallback to Subprocess if neither runtime is available

Why do we need to fallback to subprocess ? I would imagine we have 3 backend support, and user decide what they want to use:
KubernetesBackend()
ContainerBackend()
LocalProcessBackend()
In the ContainerBackend users can select:
 ContainerBackend(
  ContainerBackendConfig(container_runtime="docker")
)
or
 ContainerBackend(
  ContainerBackendConfig(container_runtime="podman")
)

Understood. Let me see what I can do. Thank you for the swift reply!

Fiona-Waters · 2025-10-08T14:19:09Z

@Fiona-Waters that sounds to me. I agree the fallback logic may really apply to choose the default container runtime.

Other than that, I'd be inclined to drop the "Local" prefix entirely. Even Kubernetes could run local with KinD, and I doubt the SDK will ever do remote process.

Ok cool. Let me see what I can do. Thank you!

Fiona-Waters · 2025-10-10T15:59:19Z

@andreyvelich @astefanutti @briangallagher
I've updated the PR. Please take a look. Thanks

astefanutti · 2025-10-15T12:25:11Z

/ok-to-test

kubeflow/trainer/backends/container/backend.py

astefanutti

@Fiona-Waters thanks for this awesome work!

That looks good to me overall.

/assign @kubeflow/kubeflow-sdk-team @briangallagher

kubeflow/trainer/backends/local_runtime_loader.py

andreyvelich

Thank you @Fiona-Waters!
I left my initial messages.

README.md

kubeflow/trainer/backends/container/README.md

kubeflow/trainer/backends/container/backend.py

andreyvelich · 2025-10-20T17:06:47Z

kubeflow/trainer/config/training_runtimes/torch_distributed.yaml

@@ -0,0 +1,25 @@
+apiVersion: trainer.kubeflow.org/v1alpha1


Instead of installing the runtimes, can we just read the image version from GitHub dynamically ?

Let me look into that. For offline support should we fall back to providing this?

@Fiona-Waters Do we require only image ?
We can fallback to the constant, that we define somewhere.

Is it okay to hardcode constants that will then need to be updated manually later? Something like this:

DEFAULT_FRAMEWORK_IMAGES = { "torch": "pytorch/pytorch:2.7.1-cuda12.8-cudnn9-runtime", }

This would then be used to create a default runtime, if the user doesn't provide a runtime url (and we have removed the sample one).

I've pushed changes to use a default image if a user doesn't provide a URL and have removed the yaml. Please review. Thanks

kubeflow/trainer/backends/container/types.py

kubeflow/trainer/backends/container/client_adapter.py

Signed-off-by: Brian Gallagher <[email protected]>

Signed-off-by: Fiona Waters <[email protected]>

… memory Signed-off-by: Fiona Waters <[email protected]>

Fiona-Waters · 2025-10-31T23:41:58Z

@andreyvelich I have addressed all comments, and rebased. Please take a look when you can. Thanks.

andreyvelich

Thanks for the updates @Fiona-Waters!
Just a few small nits, overall looks great!
We should be ready to move this forward.

kubeflow/trainer/backends/container/utils.py

kubeflow/trainer/backends/container/__init__.py

kubeflow/trainer/backends/container/adapters/__init__.py

Signed-off-by: Fiona Waters <[email protected]>

coveralls · 2025-11-01T15:28:52Z

Pull Request Test Coverage Report for Build 18998808027

Details

0 of 0 changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 79.621%

Totals
Change from base Build 18993446847:	0.0%
Covered Lines:	168
Relevant Lines:	211

💛 - Coveralls

Fiona-Waters · 2025-11-01T15:32:56Z

@andreyvelich I have addressed your most recent comments. We should be good to go now. Let me know if you want me to squash the commits. Thank you!

andreyvelich · 2025-11-01T15:34:20Z

Thanks for this great contribution @Fiona-Waters!
/lgtm
/assign @astefanutti @Electronic-Waste @kramaranya

astefanutti · 2025-11-03T11:12:33Z

Thanks @Fiona-Waters for this awesome work!

/lgtm
/approve

google-oss-prow · 2025-11-03T11:12:43Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: astefanutti

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [astefanutti]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Fiona-Waters · 2025-11-03T11:48:04Z

Thanks @andreyvelich @astefanutti, delighted to have made this contribution. Thank you, and to all reviewers for their help.
The PR for website guides and the PR for local mode notebook example can be reviewed/merged now. Thanks 🥳

google-oss-prow bot requested review from kramaranya and szaher October 7, 2025 15:03

google-oss-prow bot added the size/XXL label Oct 7, 2025

andreyvelich reviewed Oct 7, 2025

View reviewed changes

Fiona-Waters changed the title ~~feat: Add Podman backend and sync Docker backend implementation~~ [WIP] feat: Add Podman backend and sync Docker backend implementation Oct 8, 2025

google-oss-prow bot added the do-not-merge/work-in-progress label Oct 8, 2025

briangallagher mentioned this pull request Oct 8, 2025

feat: Add docker backend #111

Closed

Fiona-Waters force-pushed the podman-backend branch 4 times, most recently from bdde877 to d1288b2 Compare October 10, 2025 15:55

Fiona-Waters changed the title ~~[WIP] feat: Add Podman backend and sync Docker backend implementation~~ feat: Add Podman backend and sync Docker backend implementation Oct 10, 2025

google-oss-prow bot removed the do-not-merge/work-in-progress label Oct 10, 2025

Fiona-Waters requested a review from andreyvelich October 10, 2025 15:58

Fiona-Waters changed the title ~~feat: Add Podman backend and sync Docker backend implementation~~ feat: Add ContainerBackend with Docker and Podman Oct 10, 2025

google-oss-prow bot added the ok-to-test label Oct 15, 2025

astefanutti reviewed Oct 15, 2025

View reviewed changes

kubeflow/trainer/backends/container/backend.py Outdated Show resolved Hide resolved

Fiona-Waters force-pushed the podman-backend branch from defd7eb to 24f0bbd Compare October 20, 2025 10:40

astefanutti reviewed Oct 20, 2025

View reviewed changes

kubeflow/trainer/backends/local_runtime_loader.py Outdated Show resolved Hide resolved

andreyvelich reviewed Oct 20, 2025

View reviewed changes

briangallagher mentioned this pull request Oct 21, 2025

feat: add local docker training example kubeflow/trainer#2869

Closed

astefanutti mentioned this pull request Oct 22, 2025

Kubeflow SDK v0.2 Release #120

Open

6 tasks

Fiona-Waters force-pushed the podman-backend branch from 3e1f243 to e53ef9f Compare October 31, 2025 23:33

briangallagher and others added 5 commits October 31, 2025 23:38

Add docker backend

96f4808

Signed-off-by: Brian Gallagher <[email protected]>

Add podman backend

c436f69

Signed-off-by: Fiona Waters <[email protected]>

Implementing ContainerBackend

97ae9d2

Signed-off-by: Fiona Waters <[email protected]>

Use ip address for Podman

9a0ca37

Signed-off-by: Fiona Waters <[email protected]>

Updating to rely on container runtime rather than storing job info in…

c92f777

… memory Signed-off-by: Fiona Waters <[email protected]>

Fiona-Waters force-pushed the podman-backend branch from e53ef9f to 7931f57 Compare October 31, 2025 23:40

Fiona-Waters force-pushed the podman-backend branch 2 times, most recently from 6edf0e3 to 9bf4cb7 Compare November 1, 2025 00:38

andreyvelich reviewed Nov 1, 2025

View reviewed changes

kubeflow/trainer/backends/container/utils.py Outdated Show resolved Hide resolved

kubeflow/trainer/backends/container/__init__.py Outdated Show resolved Hide resolved

kubeflow/trainer/backends/container/adapters/__init__.py Outdated Show resolved Hide resolved

Fiona-Waters force-pushed the podman-backend branch 2 times, most recently from 5df1c72 to 51cf1ed Compare November 1, 2025 10:11

Fiona-Waters added 2 commits November 1, 2025 15:27

Addressing feedback

bf7dcf7

Signed-off-by: Fiona Waters <[email protected]>

Checking common sockets before failing

525d4c9

Signed-off-by: Fiona Waters <[email protected]>

Fiona-Waters force-pushed the podman-backend branch from 51cf1ed to 525d4c9 Compare November 1, 2025 15:27

google-oss-prow bot assigned Electronic-Waste, kramaranya and andreyvelich Nov 1, 2025

google-oss-prow bot added the lgtm label Nov 1, 2025

google-oss-prow bot added the approved label Nov 3, 2025

google-oss-prow bot merged commit 57d2052 into kubeflow:main Nov 3, 2025
13 checks passed

This was referenced Nov 3, 2025

Create Podman Backend to support Local Execution #114

Closed

Create Docker Backend to support Local Execution #108

Closed

feat: Add ContainerBackend with Docker and Podman #119

feat: Add ContainerBackend with Docker and Podman #119

Conversation

Fiona-Waters commented Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

andreyvelich left a comment

Choose a reason for hiding this comment

Uh oh!

Fiona-Waters commented Oct 8, 2025

Uh oh!

Fiona-Waters commented Oct 8, 2025

Uh oh!

andreyvelich commented Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

astefanutti commented Oct 8, 2025

Uh oh!

Fiona-Waters commented Oct 8, 2025

Uh oh!

Fiona-Waters commented Oct 8, 2025

Uh oh!

Fiona-Waters commented Oct 10, 2025

Uh oh!

astefanutti commented Oct 15, 2025

Uh oh!

Uh oh!

astefanutti left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

andreyvelich left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andreyvelich Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

Fiona-Waters Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

andreyvelich Oct 28, 2025

Choose a reason for hiding this comment

Uh oh!

Fiona-Waters Oct 29, 2025

Choose a reason for hiding this comment

Uh oh!

Fiona-Waters Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Fiona-Waters commented Oct 31, 2025

Uh oh!

andreyvelich left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

coveralls commented Nov 1, 2025

Pull Request Test Coverage Report for Build 18998808027

Details

💛 - Coveralls

Uh oh!

Fiona-Waters commented Nov 1, 2025

Uh oh!

andreyvelich commented Nov 1, 2025

Uh oh!

astefanutti commented Nov 3, 2025

Uh oh!

google-oss-prow bot commented Nov 3, 2025

Uh oh!

Uh oh!

Fiona-Waters commented Oct 7, 2025 •

edited

Loading

andreyvelich commented Oct 8, 2025 •

edited

Loading